User-adaptive restaurant management system

ABSTRACT

A user-adaptive order processing terminal includes a display, a microphone, a configuration manager, and a motion sensor. The display displays electronic menu items in a first area for selection by a user, where the electronic menu items are accessed via selection of one or more sub-menu items. The microphone detects audio spoken by the user. The configuration manager captures the audio from the microphone, transmits the audio via first messages to a backend server, receives second messages from the backend server providing text that corresponds to the audio, accesses suggested menu items that correspond to the text, and modifies a second area of the display to present the suggested menu items for selection, where the suggested menu items would otherwise be presented in the first area through selection of the one or more sub-menu items. The motion sensor subsequently detects distance to and motions performed by the user, where the configuration manager captures movements of the user from the motion sensor, transmit the motions via third messages to the backend server, receives fourth messages from the backend server providing 3-dimensional (3D) gestures that correspond to the movements, and adds one or more of the suggested menu items to an electronic order in accordance with the 3D gestures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to the following co-pending U.S. PatentApplications, each of which has a common assignee and common inventors.

SERIAL NUMBER FILING DATE TITLE ADAPTIVE RESTAURANT (TST.0113)     MANAGEMENT SYSTEM APPARATUS AND (TST.0114)      METHOD FOR ADAPTIVERESTAURANT MANAGEMENT COMMAND-ADAPTIVE (TST.0115)      RESTAURANTMANAGEMENT SYSTEM CONTEXT-ADAPTIVE (TST.0116)      RESTAURANT MANAGEMENTSYSTEM

BACKGROUND OF THE INVENTION Field of the Invention

This invention relates in general to the field of retail establishmentmanagement, and more particularly to process management systems that areadaptive for intended use and which employ one or more features directedto acceleration of order entry and fulfillment processes.

Description of the Related Art

In prior times, when a patron entered a retail establishment (e.g., astore, a restaurant, etc.), an employee of the establishment assistedthe patron in order to locate and generate an order of retail items forpreparation and delivery. The employee either remembered the items orwrote the selections down on a paper order form. The order was thencommunicated/provided to an order processing station where one or moreother employees assembled the prepared and the items on the order. Thecompleted items were then delivered to the patron. Such a system hasbeen employed for hundreds of years and is still used in many smallestablishments.

In more recent years, establishments—especially larger ones—areswitching over from paper-based ordering to electronic screen-basedordering systems. These systems may employ conventional displays forviewing menu items and conventional keyboards for entry of selected menuitems to configure an order for a patron. Other forms of electronicordering systems comprise touchscreen-based terminals (e.g., specialpurpose tablet computers—both wired and wireless—to generate orders andto communicate those orders to one or more order processing stations.These order processing stations may additionally have (typically) largertouchscreen-based terminals, whereon taken orders are displayed forpreparation, and whereby order processors indicate completion of theorders. Completion is generally indicated by touching a specified areaof the touchscreen terminals or by actuating controls that are coupledto the terminals.

As one skilled in the art will appreciate, many order processingenvironments are riddled with substances that are detrimental toelectronic devices. Consider a restaurant kitchen, where one skilledwill also appreciate that most food ingredients may damage electronicdevices when they come in contact with the devices. In addition, manyorder processing environments require that order processors wearprotective gear, which may also impede interaction with a touchscreen orother electronic device to indicate completion of orders.

The present inventors have observed the above problems and limitationsin the art and have sensed a need for apparatus and methods within orderentry and processing environments that preclude degradation ofelectronic order entry and processing devices, and that furthermore maybe employed to accelerate order entry ad processing tasks over thatwhich has heretofore been provided.

Therefore, what is needed is a retail management system that employs 3Dgestures, to accelerate placement of orders and to indicate ordercompletion.

What is also needed is a retail management system that employs voicecommands, to accelerate placement of orders and to indicate ordercompletion.

What is additionally needed is a retail management system that employsvoice commands to enter a 3D gesture mode, where the 3D gestures areused to accelerate placement of orders and to indicate order completion.

What is furthermore needed is a retail management system that employsfacial recognition for configuration of associated order entry andprocessing devices, including entering into a 3D gesture mode, where the3D gestures are used to accelerate placement of orders and to indicateorder completion.

What is moreover needed is a retail management system that suggestsorder selections based upon voice recognition of key terms, and thatutilized 3D gestures to select from among the suggested orderselections.

SUMMARY OF THE INVENTION

The present invention, among other applications, is directed to solvingthe above-noted problems and addresses other problems, disadvantages,and limitations of the prior art. In one embodiment, a user-adaptiveorder processing terminal is provided that includes a display, amicrophone, a configuration manager, and a motion sensor. The display isconfigured to display electronic menu items in a first area forselection by a user, where the electronic menu items are accessed viaselection of one or more sub-menu items. The microphone is configured todetect audio spoken by the user. The configuration manager is coupled tothe display and the microphone, and is configured to capture the audiofrom the microphone, to transmit the audio via first messages to abackend server, to receive second messages from the backend serverproviding text that corresponds to the audio, to access suggested menuitems that correspond to the text, and to modify a second area of thedisplay to present the suggested menu items for selection, where thesuggested menu items would otherwise be presented in the first areathrough selection of the one or more sub-menu items. The motion sensoris coupled to the configuration manager, and is configured tosubsequently detect distance to and motions performed by the user, wherethe configuration manager captures movements of the user from the motionsensor, transmit the motions via third messages to the backend server,receives fourth messages from the backend server providing 3-dimensional(3D) gestures that correspond to the movements, and adds one or more ofthe suggested menu items to an electronic order in accordance with the3D gestures.

One aspect of the present invention contemplates a user-adaptive orderprocessing terminal that has a display, a microphone, an orderprocessor, and a motion sensor. The display is configured to displayelectronic menu items in a first area for selection by a user, where theelectronic menu items are accessed via selection of one or more sub-menuitems. The microphone is configured to detect audio spoken by the user.The order processor is coupled to the display and the motion sensor, andis configured to maintain current states of all orders being fulfilled.The order processor includes current state fields and a configurationmanager. The current state fields each maintain a current state for acorresponding one or the orders being fulfilled. The configurationmanager is coupled to the display and the microphone, and is configuredto capture the audio from the microphone, to transmit the audio viafirst messages to a backend server, to receive second messages from thebackend server providing text that corresponds to the audio, to accesssuggested menu items that correspond to the text, and to modify a secondarea of the display to present the suggested menu items for selection,where the suggested menu items would otherwise be presented in the firstarea through selection of the one or more sub-menu items. The motionsensor is coupled to the order processor, and is configured tosubsequently detect distance to and motions performed by the user, wherethe configuration manager captures movements of the user from the motionsensor, transmit the motions via third messages to the backend server,receives fourth messages from the backend server providing 3-dimensional(3D) gestures that correspond to the movements, and adds one or more ofthe suggested menu items to an electronic order in accordance with the3D gestures.

Another aspect of the present invention comprehends a user-adaptiveorder processing method, including: displaying electronic menu items ina first area for selection by a user, where the electronic menu itemsare accessed via selection of one or more sub-menu items; employing amicrophone to detect audio spoken by the user; capturing the audio fromthe microphone, transmitting the audio via first messages to a backendserver, receiving second messages from the backend server providing textthat corresponds to the audio, accessing suggested menu items thatcorrespond to the text, and modifying a second area of the display topresent the suggested menu items for selection, where the suggested menuitems would otherwise be presented in the first area through selectionof the one or more sub-menu items; and subsequently detecting distanceto and motions performed by the user by capturing movements of the userfrom a motion sensor, transmitting the motions via third messages to thebackend server, receiving fourth messages from the backend serverproviding 3-dimensional (3D) gestures that correspond to the movements,and adding one or more of the suggested menu items to an electronicorder in accordance with the 3D gestures.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features, and advantages of the presentinvention will become better understood with regard to the followingdescription, and accompanying drawings where:

FIG. 1 is a diagram illustrating a present-day technique for restaurantmanagement that is paper based;

FIG. 2 is a diagram depicting another present-day technique forrestaurant management that is touchscreen based;

FIG. 3 is a diagram featuring a front view of an adaptive restaurantmanagement terminal according to the present invention;

FIG. 4 is a diagram showing a top view of an adaptive restaurantmanagement terminal according to the present invention;

FIG. 5 is a diagram illustrating a restaurant management terminalaccording to the present invention that provides for intelligent menuassistance;

FIG. 6 is a diagram detailing a context-adaptive terminal according tothe present invention that features an exemplary manager's display;

FIG. 7 is a diagram showing a terminal configuration screen on acontext-adaptive terminal according to the present invention;

FIG. 8 is a diagram illustrating a context-adaptive terminal accordingto the present invention that features an exemplary anti-theft display;

FIG. 9 is a block diagram depicting an adaptive restaurant managementsystem according to the present invention;

FIG. 10 is a block diagram featuring a backend server according to thepresent invention;

FIG. 11 is a block diagram showing an adaptive terminal according to thepresent invention; and

FIG. 12 is a diagram detailing exemplary update/status messagesaccording to the present invention.

DETAILED DESCRIPTION

Exemplary and illustrative embodiments of the invention are describedbelow. It should be understood at the outset that, although exemplaryembodiments are illustrated in the figures and described below, theprinciples of the present disclosure may be implemented using any numberof techniques, whether currently known or not. In the interest ofclarity, not all features of an actual implementation are described inthis specification, for those skilled in the art will appreciate that inthe development of any such actual embodiment, numerous implementationspecific decisions are made to achieve specific goals, such ascompliance with system-related and business-related constraints, whichvary from one implementation to another. Furthermore, it will beappreciated that such a development effort might be complex andtime-consuming, but would nevertheless be a routine undertaking forthose of ordinary skill in the art having the benefit of thisdisclosure. Various modifications to the preferred embodiment will beapparent to those skilled in the art, and the general principles definedherein may be applied to other embodiments. Therefore, the presentinvention is not intended to be limited to the particular embodimentsshown and described herein, but is to be accorded the widest scopeconsistent with the principles and novel features herein disclosed.

The present invention will now be described with reference to theattached figures. Various structures, systems, and devices areschematically depicted in the drawings for purposes of explanation onlyand so as to not obscure the present invention with details that arewell known to those skilled in the art. Nevertheless, the attacheddrawings are included to describe and explain illustrative examples ofthe present invention. Unless otherwise specifically noted, articlesdepicted in the drawings are not necessarily drawn to scale.

The words and phrases used herein should be understood and interpretedto have a meaning consistent with the understanding of those words andphrases by those skilled in the relevant art. No special definition of aterm or phrase (i.e., a definition that is different from the ordinaryand customary meaning as understood by those skilled in the art) isintended to be implied by consistent usage of the term or phrase herein.To the extent that a term or phrase is intended to have a specialmeaning (i.e., a meaning other than that understood by skilled artisans)such a special definition will be expressly set forth in thespecification in a definitional manner that directly and unequivocallyprovides the special definition for the term or phrase. As used in thisdisclosure, “each” refers to each member of a set, each member of asubset, each member of a group, each member of a portion, each member ofa part, etc.

Applicants note that unless the words “means for” or “step for” areexplicitly used in a particular claim, it is not intended that any ofthe appended claims or claim elements are recited in such a manner as toinvoke 35 U.S.C. § 112(f).

Definitions

Central Processing Unit (CPU): The electronic circuits (i.e.,“hardware”) that execute the instructions of a computer program (alsoknown as a “computer application” or “application”) by performingoperations on data that include arithmetic operations, logicaloperations, and input/output operations.

Microprocessor: An electronic device that functions as a CPU on a singleintegrated circuit. A microprocessor receives digital data as input,processes the data according to instructions fetched from a memory(either on-die or off-die), and generates results of operationsprescribed by the instructions as output. A general-purposemicroprocessor may be employed in a desktop, mobile, or tablet computer,and is employed for uses such as computation, text editing, multimediadisplay, and Internet browsing. A microprocessor may also be disposed inan embedded system to control a wide variety of devices includingappliances, mobile telephones, smart phones, and industrial controldevices.

Instruction Set Architecture (ISA) or Instruction Set: A part of acomputer architecture related to programming that includes data types,instructions, registers, addressing modes, memory architecture,interrupt and exception handling, and input/output. An ISA includes aspecification of the set of opcodes (i.e., machine languageinstructions), and the native commands implemented by a particular CPU.

x86-Compatible Microprocessor: A microprocessor capable of executingcomputer applications that are programmed according to the x86 ISA.

Microcode: A term employed to refer to a plurality of microinstructions. A micro instruction (also referred to as a “nativeinstruction”) is an instruction at the level that a microprocessorsub-unit executes. Exemplary sub-units include integer units, floatingpoint units, MMX units, and load/store units. For example, microinstructions are directly executed by a reduced instruction set computer(RISC) microprocessor. For a complex instruction set computer (CISC)microprocessor such as an x86-compatible microprocessor, x86instructions are translated into associated micro instructions, and theassociated micro instructions are directly executed by a sub-unit orsub-units within the CISC microprocessor.

In view of the above background discussion on retail establishmentmanagement systems and associated techniques employed within present-dayestablishments for receiving and processing orders, a discussion thesesystems will be presented with reference to FIGS. 1-2. Following this, adiscussion of the present invention will be presented with reference toFIGS. 3-12. The present invention overcomes the disadvantages andlimitations of present-day order processing techniques by providing anadaptive restaurant management system featuring hands-free kitchendisplays, kiosks, and fixed and mobile point-of-sale (POS) terminalsthat are adaptable to configuration and environment, and that providefor increased order processing throughput and efficiency, along withproviding for processing improvements in the displays, kiosks, andterminals themselves.

Referring to FIG. 1, a diagram 100 is presented illustrating apresent-day technique for restaurant management that is paper based.This technique, as one skilled in the art will appreciate, is widelyemployed within many retail establishments all over the world and ismost prevalently employed within restaurants. The diagram 100 shows anorder processor 102 who is responsible for processing orders placed forprepared goods within the establishment. In a big box store orwarehouse, the goods may comprise one or more merchandised items, andthe order processor 102 may be required to aggregate and assemble theitems. In a restaurant, the order processor 102 may be required toassemble and prepare food items selected from the restaurant's menu.

Typically, the order processor 102 may be positioned at a workstation104 and may be within sight and touch distance of one or more paperorders 108 upon which are written one or more menu items XXX, YYY, ZZZthat patrons have selected from the menu. The paper orders 108 aregenerated by staff (e.g., waiters) and brought to the workstation 104.The paper orders 108 are then temporarily affixed (e.g., clipped,pinned, etc.) to an order holding device 106 (e.g., a rail, a line, amagnetic strip, etc.) As the order processor 102 completes an assignedorder, he/she may remove its corresponding paper order 108, hand thepaper order to order delivery staff (e.g., a waiter, expeditor, etc.),and begin preparation of menu items XXX, YYY, ZZZ shown on a next paperorder 108. The diagram 100 shows the order processor 102 preparingorders from left to right as shown on the holding device 106 andremoving the leftmost paper order 108 as it is completed; however, asone skilled in the art will appreciate, some establishments work rightto left because such a convention is a function of the order processingenvironment. It is noted that to clear an order, its corresponding paperorder 108 must be touched and handled by the order processor 102 or byother staff.

Now referring to FIG. 2, a diagram 200 is presented depicting anotherpresent-day technique for restaurant management that is touchscreenbased. This technique, though not as widely employed as the paper-basedtechnique of FIG. 1, is increasingly being employed in establishmentshaving significantly greater numbers of patrons that thoseestablishments that utilize paper orders 108. The diagram 200 shows anorder processor 202 who is responsible for processing orders placed forprepared goods within the establishment. Like the scenario of FIG. 1,the order processor 202 may be required to assemble and preparemerchandise or food items selected from a menu.

Typically, the order processor 202 may be positioned at a workstation204 and may be within sight and touch distance of a kitchen displayscreen (KDS) 210 upon which are displayed one or more electronic orders208, each comprising menu items XXX, YYY, ZZZ that patrons have selectedfrom the menu. The electronic orders 208 are generated by staff (e.g.,waiters) and transmitted to the KDS 210 for display to the orderprocessor 202. In some environments, staff (e.g., waiters) may takeorders by hand or by memory and may enter those orders into a fixedorder entry terminal that is coupled to the KDS 210. In otherenvironments, staff may enter orders directly into a mobile terminalthat transmits the orders via a wireless link (e.g., Wi-Fi) to the KDS210. As one skilled in the art will appreciate, a larger establishmentmay have numerous workstations 204 and KDSs 210 for performing specifiedpreparation functions (e.g., salad station, pasta station, grill master,expeditor, etc.) and portions of a given electronic order 208 may beprepared by order processors 202 at one or more of the numerousworkstations 204.

Like the paper orders 108 of FIG. 1, the electronic orders 208 arepresented on a display area 212 of the KDS 210. In some cases, thedisplay area 212 comprises a touchscreen, and as the order processor 202completes an assigned order, he/she may remove its correspondingelectronic order 208 by performing touch gestures on the display area212 (e.g., swipe left, double-touch click, and swipe up, etc.) and beginpreparation of menu items XXX, YYY, ZZZ shown on a next electronic order208. As the electronic order 208 is removed, its completion may becommunicated to staff so that the corresponding order may be deliveredto an expeditor or directly to a patron. The diagram 200 shows the orderprocessor 202 preparing orders from left to right as shown on thedisplay 212, deleting the leftmost electronic order 208 as it iscompleted, and shifting remaining electronic orders 208 to the left;however, as noted above with reference to FIG. 1, some establishmentswork right to left, and such convention is a function of the orderprocessing environment. Like the environment of FIG. 1, to clear anorder, the order processor 202 or staff is required to physically touchthe display area 212 of the KDS 210 and, as one skilled in the art willfurther appreciate, many order processing workstations 204 presentenvironments that are detrimental to the handling of electronic devicessuch as the KDS 210. Environmental factors that may cause error,degradation, or failure of a touchscreen KDS 210 include, but are notlimited to, moisture, dirt, grease, food items, solvents, etc. Inaddition, sanitation or safety requirements may require that the orderprocessor 202 wear hand protection, which interferes with performingtouch gestures.

Accordingly, the diagram 200 shows an optional control device 214comprising one or more actuators 216. The control device 214 may becoupled to the KDS 210 via a wired or wireless link (not shown) and maybe employed by the order processor 202 in lieu of performing touchgestures on the display area 212 of the KDS 210. Generally, the controldevice 216 costs substantially less than the KDS 210. In a restaurantenvironment, the control device 214 is referred to as a bump bar 214 andcomprises actuators 216 that, when actuated, direct the KDS 210 toperform the same functions as if the order processor 202 touched thedisplay area 212. The actuators 216 may comprise mechanical switches orsmall touchscreen switches. The control device 216 may provide for somelevel of environmental resistance (e.g., switch seals, screen cover) andmay further allow for actuations by the order processor 202 when wearingprotection (e.g., gloves). It is noted, however, that a bump barconfiguration, while precluding contamination of the KDS 210, merelyshifts potential contamination effects to the bump bar 216.

The present inventors have observed that potential contamination effectsare a significant problem in the industry and have also noted that theseeffects are increasingly pronounced as more and more establishments areconverting their order processing environments from paper-basedenvironments to electronic device-based environments. In addition, thepresent inventors have observed workstation environments where it isdifficult for an order processor 202 to physically access a touchscreenKDS 210, such as environments where the KDS 210 is beyond reasonablearm's length away. Therefore, the present inventors have sensed a needin the art for more reliable, durable, and user-friendly techniques andmechanisms for processing orders in environments that present thepotential for contamination of touchscreen-based order processingdevices, such as the KDS 210 of FIG. 2.

The present invention overcomes the above noted limitations anddisadvantages, and other related problems, by providing apparatus andmethods that entirely preclude contamination of touchscreen-based orderprocessing devices, and that furthermore significantly increase orderprocessing efficiencies within order processing environments. Inaddition to these noted advantages, the techniques disclosed hereinaccording to the present invention may also be applied to related orderprocessing devices that are employed outside of a kitchen or otherhazardous environment. Such order processing devices include, but arenot limited to, handheld mobile order processing terminals,point-of-sale terminals, host terminals, and self-ordering kioskterminals. The present invention will now be discussed with reference toFIGS. 3-12.

Now referring to FIG. 3, a diagram 300 is presented featuring a frontview of an adaptive restaurant management terminal 310 according to thepresent invention. Like the diagrams 100, 200 of FIGS. 1-2, diagram 300depicts an order processor 302 who is responsible for processing ordersplaced for prepared goods within an establishment. In a big box store orwarehouse, the goods may comprise one or more merchandised items, andthe order processor 302 may be required to aggregate and assemble theitems. In a restaurant, the order processor 302 may be required toassemble and prepare food items selected from the restaurant's menu.

Preferably, the present invention is intended for use in a restaurantsetting; however, the present inventors note that the apparatus andmethods disclosed herein may be employed in any order processingenvironment, regardless of whether contaminants are present or not,because it is an objective of the present invention to not only precludephysical degradation of order processing devices, but also to improveorder processing throughput in an establishment. In order to clearlyteach aspects of the present invention, a restaurant environment will beemployed along with related terms such as waiter, host, chef, etc.,though the reader should bear in mind that the scope of the presentinvention extends beyond application in restaurants.

Accordingly, the diagram 300 depicts the adaptive terminal 310 adjacentto the order processor 302, within viewing range, but not necessarilywithin arm's reach. In one embodiment, the adaptive terminal 310 may beconfigured as a kitchen display screen (KDS) 310, that is employedwithin a restaurant kitchen to present electronic orders 308 on adisplay 314 to the order processor 302 for preparation of menu itemsXXX, YYY, ZZZ. In one embodiment, the KDS 310 comprises a touchscreendisplay 314. In another embodiment, the KDS 310 comprises a displaywithout touchscreen features. The order processor 302 may or may not beutilizing ingredients that could potentially contaminate the KDS 310.

In contrast to the KDS 210 of FIG. 2, the KDS 310 according to thepresent invention may further comprise a video camera 318, a motionsensor 320, and a microphone 316. In one embodiment, when enabled, themotion sensor 320 may be employed to detect and interpret non-touch3-dimensional (3D) gestures that are performed by the order processor302 to direct the KDS 310 to execute specified functions such as, butnot limited to, clearing an electronic order 308, scrolling to a next orprevious order 308, selecting and opening details of an order 308, andother order processing functions that may be required in a restaurantkitchen. An exemplary 3D axis is shown in the diagram 300 whereby themotion sensor 320 may interpret left to right movements as positive X,down to up movement as positive Y, and push movements as positive Z. Itis noted that this coordinate system is exemplary and provided to teachaspects of the present invention and, as one skilled in the art willappreciate, other types of coordinate systems may be employed.

In another embodiment, when enabled, the microphone 316 may be employedto detect and interpret voice commands spoken by the order processor 302or other staff (not shown) to direct the KDS 310 to execute thefunctions noted above. In a combined embodiment, when enabled, themotion sensor 320 and microphone 320 may be synergistically employed todetect and interpret both non-touch 3D gestures and voice commandsperformed and/or spoken by the order processor 302 or other staff todirect the KDS 310 to execute the functions noted above. In yet anotherembodiment, a command spoken to the microphone 316 may be employed toenable detection and interpretation of 3D gestures by the motion sensor320. In a further embodiment, a 3D gesture performed by the orderprocessor 302 may be employed to disable detection and interpretation of3D gestures by the motion sensor 320 and to enable detection andinterpretation of voice commands spoken into the microphone 316. In anadaptive embodiment, the video camera 318 may be employed to capture andimage of the order processor 302 whereby, as will be described infurther detail below, voice-based, 3D gesture-based, and video-basedfunctions of the KDS 310 are automatically configured as a function ofthe captured image.

In yet another embodiment, the KDS 310 may comprise a touchpad display310 on the order of 15 inches, an example of which is a 1I-Series 2.0for Android 15″ AiO Touchscreen as produced by Elo Touch Solutions, Inc.In yet a further embodiment, the KDS 310 may comprise a touchpad display310 on the order of 22 inches, an example of which is a 1I-Series 2.0for Android 22″ AiO Touchscreen as produced by Elo Touch Solutions, Inc.In both embodiments, the microphone 316 and camera 318 are integratedinto the touchpad displays 310, and the motion sensor 320 is coupled tothe touchpad displays 310 via a connector (not shown), such as a USBconnector. Other embodiments contemplate a camera 318, motion sensor 320and microphone 316 that are separate from, but coupled to, the touchpaddisplay 310, via a connector or other conventional port, such as a USBport.

In one embodiment, the microphone 316 may comprise a directionalmicrophone 316 with an audio field of coverage, where captured audio mayadditionally be employed by other components (not shown) of a restaurantmanagement system according to the present invention for purposes ofnoise cancellation. In another embodiment, the microphone 316 maycomprise a low-power microphone 316 that listens for one or morespecific words (e.g., “Hey KDS”), and upon detecting the one or morespecific words, commands the KDS 310 to wake up from a low-power mode,or that commands the KDS 310 to enter a 3D gesture mode.

Preferably, the motion sensor 320 may comprise one or more passiveinfrared (PIR) sensors 320 that exhibit a sensitivity range up toapproximately 20 feet with a horizontal field of view (FOV) ofapproximately 110 degrees and a vertical (FOV) of approximately 70degrees. In all embodiments, the motion sensor 320 is configured tocommunicate motions of the order processor 302.

In operation, when enabled, the motion sensor 320 may detect andinterpret a horizontal right to left motion of a hand of the orderprocessor 302 as a command to the KDS 310 complete a leftmost order 308,to communicate completion of the leftmost order 308 to staff, and tostage a next order 308 for preparation. In another embodiment, whenenabled, the motion sensor 320 may detect and interpret one or morespecified words (e.g., “KDS, clear current order”) spoken by the orderprocessor 302 as a command to the KDS 310 complete a leftmost order 308,to communicate completion of the leftmost order 308 to staff, and tostage a next order 308 for preparation. It is not a purpose of thepresent disclosure to present an exhaustive list of 3D gestures andvoice commands for use in a restaurant or other retail order processingenvironment, but rather to teach features and details that provide fortailoring of gestures, spoken audio, and captured video images within arestaurant management system according to the present invention, wherebyemployment of those tailored gestures, spoken audio, and captured videoimages result in increased order processing speeds along with increasedperformance of the order processing devices 310 themselves, withoutexposing the order processing devices 310 to any form of physicalcontact by staff 302.

Turning now to FIG. 4, diagram 400 is presented showing a top view of anadaptive restaurant management terminal 410 according to the presentinvention, such as the terminal 310 of FIG. 3. Similar to thoselike-named elements of FIG. 3, the diagram 400 depicts an orderprocessor 402 who is responsible for processing orders placed forprepared goods within an establishment, the adaptive terminal 410 thatis adjacent to the order processor 402, within viewing range, but notnecessarily within arm's reach. The adaptive terminal 410 may beconfigured in any of the embodiments discussed with reference to FIG. 3and includes a video camera 418, a motion sensor 420, and a microphone416.

The camera 418 has a focal length with an image field of view 419 thatallows for capture of images of the order processor 402 or anotherperson of interest within the image field of view 419, where the imagesare suitable for processing by facial recognition algorithms. Themicrophone 416 has a sensitivity with an audio field of coverage 417that allows for capture of speech from the order processor 402 oranother person within the field of coverage 417, where the capturedaudio quality is suitable for processing by speech-to-text algorithms.The motion sensor 420 has a sensitivity within a field of coverage 421that allows for detection and tracking of movement by the orderprocessor 402 or another person within the field of coverage 421. Inaccordance with the coordinate system shown in FIG. 3, the top view ofFIG. 4 depicts the coordinate system having perpendicular X and Z axesin the plane of the page. Thus, the Y axis is perpendicular to the X andZ axes.

The diagram 400 also shows an exemplary command table 430. Whenmovements shown in the swipe direction column of the table 430 aredetected, the restaurant management system according to the presentinvention are interpreted as corresponding commands, which are executedby the KDS 410. For example, when the motion sensor 420 detects a handswipe in the negative X direction, the restaurant management systeminterprets the movement as a command to clear a current order.Accordingly, the command is issued to and executed by the KDS 410, thuscompleting the current order and moving a next order into a currentorder position, as described above with reference to FIG. 3. Likewise, amovement in the positive Z direction results in a command to the KDS 410to initiate a cursor and begin following movement of the orderprocessor's hand with the cursor. Two movements in the positive Zdirection result in a command to the KDS 410 to select and open anorder. Movement in the negative Z direction results in deselection of anorder. Other exemplary movements are detailed in the table 430.

Now referring to FIG. 5, a diagram 500 is presented illustrating arestaurant management terminal 510 according to the present inventionthat provides for adaptive menu assistance. Like the terminals 310, 410of FIGS. 3-4, the terminal 510 in the diagram 500 may comprise a camera518, a motion sensor 520, and a microphone 516 whose configuration andoperation are substantially the same as those like-named elements inFIGS. 3-4. The terminal 510 also has a touchscreen display 514.Preferably, the terminal 510 is employed either by wait staff whenconfigured as a fixed or handheld terminal 510 for taking orders, or bypatrons when configured as a fixed or handheld self-ordering kiosk 510.Dimensions of the terminals 510 may vary according to configuration. Forexample, when configured as a handheld wireless terminal 510, theterminal 510 may comprise an approximately 8-inch Android tabletcomputer with integrated camera 518 and microphone 516, to which themotion sensor 520 is coupled via a USB port. When configured as a kiosk,the terminal 510 may comprise a 15- or 22-inch tablet as describedabove, to which the motion sensor 520 is coupled via a USB port. Otherdimensions of the terminal 510 are contemplated.

FIG. 5 is illustrates a feature of the present invention that isdirected toward acceleration of order entry by either staff or patronsover that which has heretofore been provided. As one skilled in the artwill appreciate, electronic terminals 510 typically only display aportion of a menu, primarily because the display area 514 is muchsmaller than the size required to display a full multi-page restaurantmenu. As one skilled in the art will appreciate, there are severalpresent-day techniques that provide for navigation through the menu inorder for staff/patrons to make item selections. One technique mayemploy recognizable page thumbnail images which, when selected, expandto full-size text and images. Another technique merely places full-sizetext and images in order and the staff/patrons are required to scrollthrough the text in order to locate and make menu selections. A thirdtechnique, shown in the diagram 500, employs hyperlinks (e.g., BREAKFASTMENU, LUNCH MENU, etc.) which, when selected, bring up sub-menuhyperlinks which, when selected, ultimately result in the display ofmenu items. For example, if staff/patrons select the DINNER MENUhyperlink, sub-menu hyperlinks such as APPETIZERS, ENTREES, DESSERTS,and DRINKS may be displayed. Upon further selections, ultimately menuitems (e.g., 16-Ounce Ribeye) are displayed for order entry. The presentinventors note that navigation to menu items by any of the above notedtechniques is onerous and annoying at best, but in addition adds timeand confusion to the order entry process.

Accordingly, the present invention provides for recognition andinterpretation of a limited dictionary of menu keywords viaspeech-to-text algorithms and processing within the terminal 510 toquickly navigate to those menu items or other sub-menus that present ahigh probability of being selected. Thus, the display area 514 isdivided into a primary menu display area 514.1 and an acceleratedsuggestion area 514.2. As the order process begins, the diagram 500shows the results of recognition and interpretation of the word “eggs,”and thus the restaurant system, as described in further detail below,displays two suggested menu items (CALIFORNIA SCRAMBLE and HUEVOSRANCHEROS) along with a suggested hyperlink for breakfast sides.Advantageously, accelerated menu navigation according to the presentinvention provides for significant improvements in the order entryprocess and customer satisfaction.

The present inventors also note that examples abound for acceleration ofthe process via speech-to-text and use of a limited keyword dictionary.For instance, “I don't like red wine” and “I only drink French wines,”would result in the elimination of multiple sub-menu steps, therebyquickly navigating to wine selections that are appropriate. Otherexamples that cut through multiple hyperlink traversals include:

-   -   “I want a cheeseburger and fries.”    -   “I'm gluten free.”    -   “We are interested in the specials.”    -   “Do you have a chopped salad?”

As one skilled will appreciate, the provision of recognition andinterpretation of a limited dictionary of keywords radically changes theorder processing experience, both for staff and patrons in anestablishment. In addition to the menu assist portion of the display514.2, the terminal 510 may also be configured to utilize the motionsensor 520 to detect hand/finger movements of a user to make menuselections from the suggested items in the display area 514.2. Use ofgestures, accordingly, provides advantages in that a user may place anorder via speech and/or 3D gestures, without ever coming into physicalcontact with the terminal 510. Such a configuration may be preferableduring, say, cold season or flu season, because the means fortransmission of germs are decreased in number.

While FIGS. 3-5 illustrate applications of the present invention thatemploy interpretation of voice and 3D gestures to improve the orderprocessing experience, FIGS. 6-8 show how the addition of facialrecognition techniques enable terminals—both fixed and mobile—to rapidlyadapt context for the execution of functions by a terminal to comportwith a recognized or unrecognized user.

Referring to FIG. 6, a diagram 600 detailing a context-adaptive terminal610 according to the present invention that features an exemplarymanager's display. The discussion with reference to FIGS. 3-5 focusedprimarily on utilization of 3D gestures and speech recognition tocommand a terminal 310, 410, 510 according to the present invention toperform certain actions, whether configured as a KDS 310, 410 or anorder entry terminal 510. Another object of the present invention is toenable automatic contextual configuration of a terminal 310, 410, 510 byutilizing facial recognition techniques, described in further detailbelow, to identify registered users and to configure displays andcontrol features of the terminal 310, 410, 510 in accordance with theirintended usage of the terminal 310, 410, 510.

Like the terminals 310, 410, 510 of FIGS. 3-5, the terminal 610 in thediagram 600 may comprise a camera 618, a motion sensor 620, and amicrophone 616 whose configuration and operation are substantially thesame as those like-named elements in FIGS. 3-4. The terminal 610 alsohas a touchscreen display 614. The terminal 610 may be employed eitherby wait staff when configured as a fixed or handheld terminal 610 fortaking orders, or by patrons when configured as a fixed or handheldself-ordering kiosk 610, or by order processing staff when configured asa KDS 610. Dimensions of the terminals 610 may vary according toconfiguration. For example, when configured as a handheld wirelessterminal 610, the terminal 610 may comprise an approximately 8-inchAndroid tablet computer with integrated camera 618 and microphone 616,to which the motion sensor 620 is coupled via a USB port. Whenconfigured as a kiosk or KDS, the terminal 610 may comprise a 15- or22-inch tablet as described above, to which the motion sensor 620 iscoupled via a USB port. Other dimensions of the terminal 610 arecontemplated.

In any of the above configurations, facial recognition techniques may beemployed to identify a user and to configure displays and controls thatcorrespond to the user's primary functions. In operation, when theterminal 610 is put into service, upon initialization, the terminal 610may employ the camera 618 to capture an image of the user and employfacial recognition techniques to adapt the displays and controlsprovided to comport with the user's function. In one embodiment, theimage may be captured and identified upon power up of the terminal 610.In another embodiment, touch screen options may be provided on alldisplays 314, 414, 514, 614 that, when executed, direct the terminal310, 410, 510, 610 to capture an image of the user and employ facialrecognition techniques to change the displays and controls previouslyprovided to comport with the user's function.

In one embodiment, the displays and controls of the terminal 310, 410may result from recognition of the order processor 302, 402 along withtheir function. Accordingly, upon initialization, the KDS 310, 410captures and identifies an image of the order processor 302, andconfigures the terminal 310, 410 as a KDS terminal 310, 410. Based uponthe role of the order processor 302, 402 (e.g., grill master, saladstation, pasta station, etc.), the KDS 310, 410 may further tailor menuentries XXX, YYY, ZZZ on the electronic orders 308 to comport with theuser's role. For example, if menu item XXX corresponds to a salad, andthe user 302, 402 is identified as a grill master, then the menu itemXXX is not displayed; only menu items corresponding to work required ofthe grill master are displayed. Likewise, 3D gestures and/or voicecontrols may be enabled or disabled in accordance with preferences thatare preconfigured for employment of the terminal 310, 410 according tothe role of the user 302, 402.

In similar manner, facial identification of another user may result inthe terminal 510 as discussed with reference to FIG. 5 (including voicecontrols), when that user is recognized in their role as a waiter.

The diagram 600 shows a display/control configuration resulting fromrecognition of a captured image of an employee having managementfunctions and privileges. Accordingly, upon initialization (or viadirect command), the terminal 610 directs the camera 618 to capture andrecognize the user's image, which is shown in display area 614.3. Area614.1 shows the user's credential's and area 614.2 shows an exemplarymanager's display with hyperlinks to functions routinely performed by amanager such as reviewing revenue reports, inventory reports, accountspayable, personnel records, etc. Additionally, presuming that onlymanagers are allowed to configure a terminal 310, 410, 510, 610, thearea 614.2 also shows a hyperlink to a terminal configuration functionwhich, when selected, results in the terminal configuration shown inFIG. 7. As with all of the terminals 310, 410, 510 discussed above, theterminal 610 shown in the diagram may be configured to employ speechand/or 3D gestures as well to accelerate performance of managerialfunctions.

Turning now to FIG. 7, a diagram 700 is presented showing a terminalconfiguration screen on a context-adaptive terminal 710 according to thepresent invention. As noted above, such an exemplary display 714 resultsfrom a manager executing the TERMINAL CONFIGURATION hyperlink shown inFIG. 6. Like the terminals 310, 410, 510, 610 of FIGS. 3-6, the terminal710 in the diagram 700 may comprise a camera 718, a motion sensor 720,and a microphone 716 whose configuration and operation are substantiallythe same as those like-named elements in FIGS. 3-6. The terminal 710also has a touchscreen display 714. Accordingly, the manager's image isshown in display area 714.3, the manager's credentials are shown indisplay area 714.1, and a terminal confirmation form is shown in displayarea 714.2. Therein, the manager may provide information forconfiguration of any of the terminals 310, 410, 510, 610, 710 used inthe establishment. Thereafter, the terminals 310, 410, 510, 610, 710 mayreceive the information as will be discussed below and reconfigure uponinitialization or direct reconfiguration command. The exemplary displayarea 714.2 allows the manager to enter a terminal identification number(shown as F2742. Likewise, terminal F2742 may be configured to operatein a fixed or mobile mode, and default function of terminal F2742 may beconfigured as a host terminal (used in restaurants to route patrons totables and to assign wait staff for service), a point-of-sale (POS)terminal (used for order entry and payment), a KDS (used by an orderprocessor to prepare orders), or a kiosk (a terminal whereby patronsdirectly place orders). Terminal F2742 may further be configured toutilize any combination of 3D gestures, facial recognition, and voicecommands. When 3D gestures (“MOTION”) are enabled, Terminal F2742 mayfurthermore be configured to adjust the size of text displayed to theuser as a function of the user's distance from the terminal, asdetermined by the motion sensor. When voice commands are enabled,terminal F2742 may moreover be configured as a primary input for voicecommands, as a background audio source that is employed for noisecancellation, or both as a primary input for voice commands and abackground audio source. Clearly, as one skilled in the art willappreciate, configuration of any of the terminals 310, 410, 510, 610,710 discussed herein may involve other settings as well, and theterminals 310, 410, 510, 610, 710 may have configuration that isinitialized by default.

Though terminals 310, 410, 510, 610, 710 may be configured to employfacial recognition identify authorized users and registered patrons,another adaptive configuration contemplates detection of unauthorizedusers and adaptation of functions of the terminals 310, 410, 510, 610,710 to preclude damage and theft. A terminal 810 thus configured isshown in the diagram 800 in FIG. 8. Like the terminals 310, 410, 510,610, 710 of FIGS. 3-7, the terminal 810 in the diagram 800 may comprisea camera 818, a motion sensor 820, and a microphone 816 whoseconfiguration and operation are substantially the same as thoselike-named elements in FIGS. 3-7. The terminal 8710 also has atouchscreen display 814. Recall that when a terminal 310, 410, 510, 610,710 is initialized or directly commanded, the camera 818 captures andimage of a user. If the user is recognized, the terminal 810 isconfigured to function according to the user's role. However, if theuser is not recognized, the configuration of the terminal 810 mayautomatically adapted to any of a number of other modes. For instance,if the user is not recognized during operating hours, the terminal 810may just temporarily disable voice command or 3D gestures and mayfurthermore disable input to the terminal via touch. The diagram 800show an exemplary configuration that may be employed when theestablishment is closed for purposes of precluding tampering or theft.Accordingly, in this mode, the motion sensor 820 may be enabled and,when movement is detected, the terminal 810 may direct the camera 818 tocapture one or more images of an unauthorized user, which is shown indisplay area 814.3. Display area 814.1 may be employed to show a message(UNAUTHORIZED ACCESS) to get the attention of the unauthorized user, anddisplay are 814.2 may be employed to flash a message to the unauthorizeduser to preclude tampering and/or theft. As will be discussed in moredetail below, upon detection of the unauthorized user, other terminalswithin the establishment may be configured to perform related securityfunctions such as, but not limited to, recording stills or video,tracing movement, flashing messages, recording audio, etc.Advantageously, the adaptive terminals 310, 410, 510, 610, 710, 810according to the present invention provide for functions substantiallybeyond mere processing of order, resulting in safer, more secure workenvironments.

Although elements and operation of the terminals 310, 410, 510, 610,710, 810 has been discussed above, a primary objective of thediscussions with reference to FIGS. 3-8 has been to teach how adaptiveterminals according to the present invention may be used under a numberof different configurations. Attention is now directed to FIGS. 9-12where design details of an adaptive restaurant management systemaccording to the present invention are now presented.

Referring now to FIG. 9, a block diagram is presented depicting anadaptive restaurant management system 900 according to the presentinvention. The system 900 may include one or more service areas 902,such as a front service area 902, service area 1-service area N 902, andservice area kitchen 902. As alluded to above, the present inventorsreiterate that though the present invention is applicable to any type ofretail establishment, a restaurant establishment is herein primarilyemployed in order to teach relevant aspects of the present invention.

The service areas 902 may comprise one or more wireless access points901. The service areas 902 may also comprise one or more wireless POSterminals 911, 921, 931, coupled to the access points 901 via wirelesslinks 903, and which are distinguished in the system 900 as a hostterminal 911, mobile terminals 921, and order processing terminals 931(also referred to as KDSs 931). The service areas 902 may furthercomprise a gateway 913 to which is coupled one or more fixed hardwiredterminals 912, and which provides for coupling of the fixed terminals912 and access points 901 to an internet cloud 960 via conventionalwired links 905 such as, but not limited to, Ethernet, cable, fiberoptic, and digital subscriber line (DSL) As part of the network path toand through the cloud 960, providers of internet connectivity (e.g.,ISPs) may employ wireless technologies from tower to tower, etc., butfor purposes of this application, such links 905 will be referred to asconventional wired links 905 to distinguish them from local and cellularwireless links 901. The wireless links 903 may comprise, but are notlimited to, Wi-Fi, Bluetooth, near field communications, infrared links,IEEE 802.15.4, Zigbee radio links, and cellular based links (e.g., 3G,4G, LTE) or a combination of the noted links. The POS terminals 911,912, 921, 931 may be adaptively configured to comport with intendedfunction (i.e., host seating, order and payment entry, orderfulfillment, etc.) as described above with reference to FIGS. 3-8, andtheir assistive features (e.g., facial recognition, voice commands,motion sensing, keyword recognition and menu assist) may beenabled/disabled via any one or more of the techniques discussed abovewith reference to FIGS. 3-8. In one embodiment, the mobile terminals 921may comprise a touch screen display, camera, motion sensor, microphone,and integral payment processor (e.g., card/chip reader) that providesfor both order entry, display of order status, and payment processing.As such, the host terminal 911, fixed terminals 912, and KDSs 931 maycomprise larger touch screens to allow for easier viewing by restaurantstaff along with camera, motion sensor, and microphone, or they maycomprise displays with keyboard entry and separately connected camera,motion sensor, and microphone. In one embodiment, terminals 911, 912,931 may comprise desktop computers, laptop computers, smartphones, ortablets that are executing application programs or web-enabledapplication programs that provide for communication with a backgroundserver 970 for purposes of order entry, status updates, configuration,facial recognition, speech-to-text processing and, optionally, paymentprocessing.

The background server 970 is coupled to the internet cloud 260, and toan administrative console 971 via a conventional wired link 905 and/or awireless link 903. The background server 970 is not on-premise and isthus also referred to herein as a cloud server 970. The administrativeconsole 971 may be disposed within the restaurant premises and coupledto the background server 970 via the links 903, 905, or the console 971may be disposed in another location, say, at an operations headquartersfor multiple restaurants within a given region. In addition, the system900 may comprise one or more browser-based terminals 981 that arecoupled to the background server 970 via links 905. In one embodiment,the browser-based terminals 981 may comprise desktop computers, laptop,computers, smartphones, or tablets that are running stand-aloneapplications or web-enabled applications that provide for communicationwith the background server 970 for purposes of order entry, statusupdates, and optionally, payment processing.

The system 900 may further comprise one or more third-party-basedterminals 941 that are coupled to the background server 970 via theconventional links 905 though the cloud 960. The third party-basedterminals 941 may comprise desktop computers, laptop, computers,smartphones, or tablets that are running stand-alone third-partyapplications or web-enabled third-party applications that provide forcommunication with the background server 970 for purposes of orderentry, status updates, and optionally, payment processing via aproprietary application programming interface (API) 942. An example ofsuch a terminal 941 may include the well-known GrubHub third-partyapplication that is configured to communicate with the background server970 via the API 941.

The system 900 may further comprise one or more delivery terminals 951that are coupled to one or more cellular access points 901 viaconventional cellular wireless links 903, and the cellular access points901 are coupled to the background server 970 via the cloud 960. In oneembodiment, the delivery terminals 951 are substantially similar to themobile terminals 921 in form and function, and that are configured toprovide services for order entry, status updates, order fulfillment(i.e., delivery), and payment processing. In another embodiment, thedelivery terminals 951 are disposed as smartphone or tablets withintegrated camera and microphone, and with detachable motion sensor andpayment processor (e.g., card/chip reader). In a further embodiment, thedelivery terminals 951 are disposed as smartphone or tablets withcamera, microphone, motion sensor, and payment processor integratedwithin a single housing. Other embodiments are contemplated.

Service areas 902 corresponding to the mobile terminals 921 may have oneor more tables 904 corresponding to one or more orders. For clarity,service area 1 902 depicts two tables 904, one of which corresponds toorder 1 OD1, and the other of which corresponds to order 2 OD2. Themobile terminals 921 within service area 1 902 may processes portions ofboth orders 1 OD1 and order 2 OD2.

Service area N 902 depicts two tables 904, both of which correspond toorder A ODA. The mobile terminals 921 within service area N 902 may bothprocess portions order A ODA.

Though disposed within separate service areas (service area 1902-service area N 902), the mobile terminals 921 therein may be furtherconfigured to process portions of any and all orders within therestaurant and may roam from service area 902 to service area 902 tosupport work load of the restaurant.

The order processing terminals 931 may process all orders in therestaurant, or they may be configured to each process a portion of allof the orders in the restaurant according to their role (e.g., saladpreparation station, fry cook station, inventory station, etc.).

The host terminal 911 and fixed terminals 912 may be configured toprocess all orders in the restaurant to provide for on-premise seatingassignment, order initiation, order selection, and payment processing,including closeout of orders.

One or more restaurant staff members (not shown) within service area 1902-service area N 902 may have a personal device (e.g., smartphone,tablet, laptop) 906 that can provide an ad hoc network (i.e., hotspot)to which one or more of the mobile terminals 921 may tether for purposesof communicating with the backend server 970 in the absence of Wi-Ficonnectivity to the access points 901.

In one embodiment, operations are initiated when the one or more patronsenter the restaurant. Generally, a host (not shown) will create an order(along with corresponding order identifier (OID) via the host terminal911 for the one or more patrons and will seat the patrons at one or moretables 904. The created order may include service area designation andassignment of the order to one or more mobile terminals 921. In anotherembodiment, mobile terminals 921 within a service area 902 are assignedto all orders within that service area 902. Other embodiments arecontemplated. The created order and service area assignment aretransmitted over the cloud 960 to the backend server 970, whichmaintains durable terminal queues within which are stored order updatesfor all orders in the restaurant. Each of the durable queues correspondto each of the terminals 911, 912, 921, 931, 951 within the system 900.When connection status to a given terminal 911, 912, 921, 931, 951 isdown (i.e., the server 970 cannot verify communication with the giventerminal 911, 912, 921, 931, 951), then the server 970 maintains theorder updates for that terminal 911, 912, 921, 931, 951 untilconnectivity is reestablished, at which time the server 970 may transmitone or more of the order updates to the terminal 911, 912, 921, 931,951, verifying with each transmission that the terminal 911, 912, 921,931, 951 received the update. Advantageously, each of the terminals 911,912, 921, 931, 951 is capable of processing portions of any of theorders in the restaurant.

Likewise, each of the terminals 911, 912, 921, 931, 951 maintainsdurable order queues within which are stored order updates only for eachof the orders being processed by the terminal 911, 912, 921, 931, 951.Each of the terminals 911, 912, 921, 931, 951 also maintains a pluralityof order states that depict a current state for each of the orders inthe restaurant. As a seated patron selects one or more menu items, waitstaff enters the menu items as an update in one of the terminals 911,912, 921, 931, 951, generally a mobile terminal 921 assigned to thegiven service area 902. The order update is entered into one of thedurable order queues that corresponds to the order ID. If connectivityif present, then the terminal 911, 912, 921, 931, 951 transmits theorder update to the server 970 and waits for the server 970 toacknowledge the order update. If acknowledged, the terminal 911, 912,921, 931, 951 removes the order update from the one of the durable orderqueues. If unacknowledged (i.e., in the case of non-persistent networkconnectivity), the terminal 911, 912, 921, 931, 951 maintains the orderupdate in the one of the durable order queues until such time asconnectivity is reestablished, and the terminal 911, 912, 921, 931, 951completes transmission of the order update with acknowledgement by theserver 970.

Upon reception of a particular update from the server 970, the terminals911, 912, 921, 931, 951 may check one of their plurality of order statesthat correspond to the particular update for conflicts, as will bedescribed in further detail below. If a conflict exists, the terminals911, 912, 921, 931, 951 may utilize domain specific rules to resolve theconflict in order to establish a valid order state. Each of the 911,912, 921, 931, 951 is configured with the same domain specific rules toprovide for consistent resolution of order states.

As patrons continue to order items corresponding to the order ID, theone or more of the terminals 911, 912, 921, 931, 951 may enter the orderupdates and transmit/durably queue the order updates to the server 970in accordance with connectivity conditions. The server 970 may alsoqueue/transmit order updates for all orders in the restaurant to each ofthe terminals 911, 912, 921, 931, 951 according each terminal'sconnectivity. Order fulfillment, payment, and closeout are likewisehandled as order updates through the server 970 and arequeued/transmitted to all of the terminals 911, 912, 921, 931, 951 inaccordance with the connection status of each terminal. Deliveryterminals 951 may not be on premise at the time an order is placed;however, the system 900 according to the present invention may treat thedelivery terminals 951 as if they are virtually on premise, assigningthem to a virtual on-premise service area 902.

Patrons outside of the restaurant are also handled in similar fashionvia the browser-based terminals 981, and third-party terminals 941,though without feedback from the server 970 regarding all orders in therestaurant. When accessed through the browser-based terminals 981 andthird-party terminals 941, the server 970 creates and order ID andassigns it to one of the order processing terminals 931 for fulfillment,while sending status updates on the order ID to all of the terminals911, 912, 921, 931, 951 via the durable terminal queues therein. Theserver 970 may designate a specific delivery terminal 951 for pickup,delivery, and payment based upon proximity to the restaurant, or basedupon workload corresponding to the delivery terminal 951.

The administrative console 971 may maintain a master record of all orderstates and order updates according to all of the terminals 911, 912,921, 931, 951 in order to provide for restaurant management,maintenance, analytics, and network traffic analyses. The console 971may alternatively be disposed in an expediter's area of the restaurantfor use by expediters in assignment and allocation of patron seating andterminals 911, 912, 921, 931, 951.

The durable terminal queues and durable order queues may be disposed asnon-transitory battery backed random-access memory,electrically-erasable programmable read-only memory, solid state memory,hard disk memory, or a combination of the above that will provide formaintaining order updates within the queues across network and powerinterruptions.

Advantageously, the present invention provides for more efficientperformance of computational resources within the server 970 and theterminals 211, 911, 912, 921, 931, 951 over that which has heretoforebeen provided because multiple terminals 911, 912, 921, 931, 951 may beassigned to process portions of a single order, resulting in more timelyprocessing of the single order. Similarly, any of the terminals 911,912, 921, 931, 951 in the restaurant may be immediately reassigned to aparticular order to replace a malfunctioning terminal or to increasethroughput of the server 970. Likewise, each of the terminals 911, 912,921, 931, 951 may be configured as disclosed above with reference toFIGS. 3-8 to adapt to operational context of a recognized (orunrecognized) user, and they may furthermore be enabled to accelerateorder entry and processing through the employment of facial recognition,3D gestures, or voice commands, or a combination of facial recognition,3D gestures, and voice commands. The terminals 911, 912, 921, 931, 951,may further be configured to accelerate order entry and processing byperforming voice recognition of selected menu keywords, which providefor high probability suggestions for menu items, as is discussed abovewith reference to FIG. 5. Accordingly, computational resources 911, 912,921, 931, 951, 970 within the system 900 are afforded an overallperformance improvement as a result of acceleration of the order entryand processing process, elimination of tedious drill-down sub-menuhyperlinks, recognition of users and patrons along with automaticadaptation of functions according to role. Reliability andmaintainability of terminals 911, 912, 921, 931, 951 for use in areashaving conditions that are detrimental is increased via enablement ofnon-touch 3D gestures.

Now turning to FIG. 10, a block diagram is presented featuring a backendserver 1000 according to the present invention, substantially similar tothe server 970 of FIG. 9. The server 1000 may comprise communicationscircuitry COMMS 1002 (e.g., transceivers, modems, message formatter,etc.) that is coupled to one or more wired or wireless communicationslinks 1001, examples of which are described above with reference to FIG.9. The server 1000 may also comprise a terminal status element 1005, aterminal update element 1006, a facial processor 1008, a paymentprocessor 1004, and a speech processor 1024, all of which are coupled toCOMMS 1002 via a message bus MSG. The terminal status element 1005 iscoupled to the terminal update element 1006 via a status bus STS. Theterminal update element 1006 may comprise a service area map SA MAP 1007and a terminal configuration table 1009. The terminal update element1006 is coupled to the facial processor 1008, the payment processor1004, the speech processor 1024, a motion processor 1030, and an orderinitiation element 1003 via a terminal bus TBUS. The terminal updateelement 1006 is also coupled to a queue processor 1010 via a queue busQBUS.

The queue processor 1010 may include a durable terminal queue 1011 thatincludes terminal update records 1012, each of which are associated witha corresponding terminal, such as the terminals 911, 912, 921, 931, 951discussed with reference to FIG. 9, that is employed within a givenrestaurant. In the embodiment of FIG. 10, N terminal update records 1012are shown, each associated with a corresponding one of N terminals forthe given restaurant. In a large restaurant or big box environment, Nmay be roughly equal to 100 terminals, though larger and smaller numbersare contemplated.

Each of the terminal update records 1012 may comprise update fields1013, which are employed to queue order updates for transmission to eachof the corresponding terminals as connectivity to the correspondingterminals permits. Update fields 1013 nearest to OUT are the oldestorder updates queued for transmission to the corresponding terminals.Update fields 1013 nearest to IN are youngest (or most recent) orderupdates queued for transmission to the corresponding terminals. Fields1013 between the oldest order updates and the youngest order updatesdescend in age from oldest to youngest update according to when thoseupdates are received from others of the corresponding terminals.

Values of the order update fields 1013 may include, but are not limitedto, an order ID along with order details taken by the others of thecorresponding terminals. Accordingly, the terminal update record 1012for terminal 1 TERM1 depicts a plurality of order update fields 1013 tobe transmitted to TERM1 when connectivity is reestablished with TERM1.In decreasing age from oldest to youngest order update, the fields 1013depict updates to order 64 U64, then order 6 U6, then order 22 U22, andso on, culminating with an update to order 17 U17. As one skilled in theart will appreciate, the terminal update record 1012 for TERM1 isindicative that TERM1 has been offline (i.e., no connectivity) longerthan any of the other terminals in the restaurant. This length of timemay correspond to a mobile terminal that is serving a party on arestaurant porch that has poor Wi-Fi connectivity, or may correspond toa delivery terminal that is traversing an area with poor cellularcoverage. The terminal update records 1012 corresponding to TERM2, TERM3, and TERMN depict a number of populated order update fields 1013 lessthan the number of fields for TERM1, which may correspond to terminalswithin the restaurant that have only slightly intermittent connectivity.And the terminal update record for TERM4 contains only empty orderupdate fields 1013, thus indicated that this terminal is up to date onall order state changes within the restaurant.

Operationally, the terminal status element 1005 may periodicallytransmit a first message to each of the terminals and update theconnectivity status of the terminals based upon whether they acknowledgethe first message or not. In one embodiment, the first message maycomprise a ping message. In one embodiment, acknowledgment may comprisea simple acknowledge message. In other embodiments, acknowledgement maycomprise additional data such as received signal strength indicationRSSI, number of hops, or Global Positioning System (GPS) coordinates.The acknowledge message may further comprise configurationidentification data such as mode (e.g., mobile order entry, KDS, kiosk,management terminal) and order accelerator features (e.g., 3D gestures,voice commands, facial recognition, menu assist, etc.).

The terminal status element 1005 may provide connectivity status of eachof the terminals to the terminal update element 1006 via bus STS. Theservice area map 1007 is a table that associates each of the terminalsto one or more service areas within the restaurant. In one embodiment,the terminal update element 1006 may generate order update messages fromoldest to youngest update for each of the terminals that are connected.Connectivity is maintained when a terminal acknowledges receipt of anorder update message. Once acknowledged, the terminal update element1006 directs the queue processor 1010 to delete the oldest order updatefor that terminal and shift pending order updates so that the nextoldest order update becomes the oldest order update. In one embodiment,order updates are transmitted to a given terminal until its terminalupdate record 1012 is empty, or until connectivity is broken.

In one embodiment, all of the terminals associated with the restaurantare updated by the terminal update element 1006. In an alternativeembodiment, terminals are selectively updated in accordance with theirmapping to the one or more service areas, as indicated by contents ofthe service area map 1007. For example, delivery terminals may onlyrequire knowledge of orders that are to be delivered outside therestaurant, and thus they may be mapped to a “delivery” service area sothat order updates that correspond to the delivery service area aretransmitted to the delivery terminals. Similarly, the restaurant orretail establishment may be so large that management dedicates certainterminals to designate service areas. Accordingly, all of the terminalsin a given service area may be employed to update any order placedwithin the given service area, but they may not be employed to updateorders placed outside of the given service area.

The terminal update element 1006 may also include a terminalconfiguration table 1009, which contains records for each of theterminals in the system 1000. The records indicate a defaultconfiguration for each of the terminals along with their currentconfiguration and features, the terminal configuration table 1009 may beaccessed as required other elements of the system over TBUS. Forterminals that employ 3D gestures, the terminal configuration table 1009also provides a mapping of 3D gestures to terminal commands. Forexample, the table 1009 may indicate that a swipe left results in aterminal command to complete a current order and display a next order.

Messages received from the communications circuit 1002 may also requireadditional functions to be performed by the backend server 1000. Forexample, when orders are placed by a browser-based or third-party basedterminal, the terminal update element 1006 may transmit the order updateto the order initiation element 1003 via TBUS. The order initiationelement 1003 may then create an order ID for the order update and mayassign the order ID to one or more of the terminals within therestaurant. Similarly, when an order update message received over theCOMMS 1002 requires processing of transactions outside of the terminals'capabilities (e.g., financial transactions with credit card providers,loyalty card discounts, etc.), the payment processor 1004 may generatemessages to complete the transactions and the messages are transmittedvia COMMS 1002. The payment processor 1004 may further generate orderupdates (e.g., “order paid,” “payment source 1 approved,” “discountamount,” etc.) to be transmitted to the terminals and may provide theseupdates to the terminal update element 1006 via TBUS. The terminalupdate element 1006 may then provide those updates to the durable queue1011 via QBUS, and the updates are transmitted to the POS terminals indue course dependent upon connection status, as is described above.

Messages received from the communications circuit 1002 may comprisecaptured images or electronic representations thereof, which are routedto the facial processor 1008. The facial processor 1008 may comprise afacial ID table 1022 that comprises a mapping of captured images toauthorized users and registered patrons as discussed above. The facialID table 1022 may also include a default terminal configuration,including adaptive features (e.g., 3D, voice commands, menu assist) foreach of the authorized users. Accordingly, the facial processor 1008 maygenerate messages designated for one or more terminals that correspondto a given captured image, indicating recognition or non-recognition ofa user/patron corresponding to the captured image. The facial processor1008 may further generate messages to designated terminals thatcorrespond to a recognized user that directs the designated terminals tochange their configuration to comport with user requirements (e.g.,change to KDS configuration, enable voice commands, etc.). Portions ofthe facial processor 1008 may alternatively utilize internet-based 3rdparty facial recognition services, which are accessed and employed viamessages generated by the facial processor 1008 and provided to COMMS1002 via MSG. Accordingly, when a captured image is received, the facialprocessor 1008 may generate messages over MSG that are transmitted tothe 3rd party facial recognition services that include the capturedimage and requesting identification. The facial processor 1008 mayfurther receive messages from the 3rd party facial recognition servicesthat designate an ID that corresponds to the captured image. The ID isthen employed as an index into the facial ID table 1022 to retrieve theuser's default terminal configuration. The facial processor 1008 mayadditionally generate messages to one or more designated terminals thatinclude the default configuration.

Messages received from the communications circuit 1002 may additionallycomprise audio that has been captured by one or more terminals, whichare routed to the speech processor 1024. The speech processor 1024 thenprovides the audio to a speech-to-text element 1027 and optionally to anoise cancellation element 1028. If the audio corresponding to the oneor more terminals is to be employed for noise cancellation purposes, thenoise cancellation element 1028 may subtract the audio from audioreceived from surrounding terminals prior to processing by thespeech-to-text element 1027. The speech-to-text element 1027 convertsaudio signals/files to text signals/files and provides the textsignals/files to a keyword table 1026. The keyword table 1026 is adictionary whose contents are limited to voice commands and menu itemsthat are employed within the restaurant system. Keywords that correspondto converted audio are then incorporated into messages generated by thespeech processor 1024, which are then transmitted to a designatedterminal via MSG.

The speech processor may alternatively employ an internet-based 3rdparty speech-to-text service, which is accessed via messagesgenerated/received over MSG. In this embodiment, audio signals/filesthat have been converted to text signals/files are received over MSG andused to index the keyword table 1026, and keywords that correspond toconverted audio are then incorporated into messages generated by thespeech processor 1024, which are then transmitted to a designatedterminal via MSG.

Messages received and transmitted over MSG may further comprise capturedmotion from designated terminals. The messages are provided to a motionprocessor 1030 that includes a global gesture table 1032 that provides amapping of captured motion to a global set of 3D gestures, such as, butnot limited to, swipe left, swipe right, swipe up, swipe down, tap, tapand hold for cursor, double tap, reverse tap, etc. Captured motion thatcorresponds to 3D gestures is then transmitted to the designatedterminals via messages over MSG. In one embodiment, all motion receivedover motion sensors in the restaurant system is processed by the motionprocessor 1032. In an alternative embodiment, one or more of theterminals themselves process motion detected by their correspondingmotion sensors.

The backend server 1000 according to the present invention is configuredto perform the functions and operations as discussed above. The server1000 may comprise digital and/or analog logic, circuits, devices, ormicrocode (i.e., micro instructions or native instructions), or acombination of logic, circuits, devices, or microcode, or equivalentelements that are employed to execute the functions and operationsaccording to the present invention as noted. The elements employed toaccomplish these operations and functions within the server 1000 may beshared with other circuits, microcode, etc., that are employed toperform other functions and/or operations within the server 1000.According to the scope of the present application, microcode is a termemployed to refer to a plurality of micro instructions. A microinstruction (also referred to as a native instruction) is an instructionat the level that a unit executes. For example, micro instructions aredirectly executed by a reduced instruction set computer (RISC)microprocessor. For a complex instruction set computer (CISC)microprocessor such as an x86-compatible microprocessor, x86instructions are translated into associated micro instructions, and theassociated micro instructions are directly executed by a unit or unitswithin the CISC microprocessor.

The backend server 1000 may be embodied as one or more centralprocessing units (CPUs) that are coupled to both transitory andnon-transitory storage (i.e., memory), where one or more applicationprograms that are configured to perform the server functions discussedabove are stored in the non-transitory storage, transferred to thetransitory storage at run time, and executed by the one or more CPUs.

FIG. 11 is a block diagram showing an adaptive terminal 1100 accordingto the present invention, as discussed above with reference to FIGS.3-10. The terminal 1100 may comprise a communications circuit COMMS 1102(e.g., transceivers, modems, message formatter, etc.) that is coupled toone or more wired or wireless communications links 1101, examples ofwhich are described above with reference to FIG. 9-10. The terminal 1100may also comprise a connection monitor 1104, an order processor 1110, aand a payment processor 1106, all of which are coupled to COMMS 302 viaa message bus MSG. The terminal 1100 may also comprise a link selectelement 1105 that is coupled to the connection monitor via bus CS and tothe COMMS 1102 via bus LNK. The order processor 1110 is coupled to theconnection monitor 1104 via bus CBUS and to the payment processor 1106,an order initiation element 1107, a local gesture table 1131, a GPSreceiver 1108, and a local keyword table 1132 via bus SBUS. The orderprocessor 1110 is coupled to a touchscreen display 1103 via bus DATA, toa microphone 1113 via bus AUDIO, to a camera 1114 via bus VIDEO, to amotion sensor 1115 via bus SENSE, and to a terminal ID element 1109 viabus TID. The order processor 1110 is also coupled to a state processor1120 via a queue bus QBUS.

The state processor 1120 may include a durable order update queue 1121that includes order update records 1122, each of which are associatedwith a corresponding order for the terminal 400. Individual terminals400 are identified by their corresponding terminal ID, which may bestored within and accessed from the terminal ID element 1109. Theterminal ID element 1109 also is employed for storage and retrieval ofterminal configuration information (e.g., KDS, kiosk, mobile terminal,etc.) along with enablement/disablement of order processing accelerationfeatures (i.e., voice commands, 3D gestures, menu assist, facialrecognition).

Each of the order update records 1122 may comprise order state fields1123, which are employed to queue order state changes (i.e., orderupdates) for transmission to a synchronization server (not shown) asconnectivity to the backend server permits. State fields 1123 nearest toOUT are the oldest order state changes queued for transmission to thebackend server. State fields 1123 nearest to IN are youngest (or mostrecent) order state changes queued for transmission to the backendserver. Fields 1123 between the oldest state fields 1123 and theyoungest state fields 1123 descend in age from oldest to youngest orderstate change according to when those state changes are entered by theterminal 1100.

Values of the order state fields 1123 may include, but are not limitedto, an order ID along with order details taken by the terminal 1100.Accordingly, the order update record 1122 for order 27 O27 depicts aplurality of order state fields 1123 to be transmitted to the serverwhen connectivity is reestablished. In decreasing age from oldest toyoungest order state change, the fields 123 depict order state changes51 through SN. As one skilled in the art will appreciate, the orderupdate record 422 O27 depicts that many more state changes have beenentered while connection status of the terminal 1100 is down than havebeen entered for orders 62 O62 through order 3 O3. Advantageously, theterminal 1100 according to the present invention may be employed forentry of order updates even in the presence of network interruptions,which is characteristic of most Wi-Fi networks.

In operation, order state changes result from two sources: thetouchscreen display 1103 and messages received over COMMS 1102 from thebackend server. In the first case, wait staff may enter menu items asrequested by patrons, or in the case that the terminal 1100 isconfigured as a kiosk 1100, the patrons may enter the order itemsthemselves. The present invention contemplates provisions within theterminal 1100 to display menu selections and payment options to bothwait staff and patrons. Order items received from the touchscreendisplay 1103 are provided to the order processor 1110 via bus DATA,which generates the state changes. State changes received from thebackend server are provided to the order processor 1110 in messages overbus MSG.

The connection monitor 1104 may monitor reception of a first message(e.g., a ping message) from the backend server and direct transmissionof an acknowledgement message. The connection monitor 1104 may updatethe connectivity status of the terminal 1100 accordingly. In oneembodiment, acknowledgment may comprise a simple acknowledge message. Inother embodiments, acknowledgement may comprise additional data such asreceived signal strength indication RSSI associated with one or moreaccess points, number of hops between the backend server and theterminal 400, or Global Positioning System (GPS) coordinates, as will bedescribed in further detail below.

The link select element 1105 may be employed to direct the COMMS 1102 tochange links 1101 over which to communicate with the backend server,such as switching from Wi-Fi to LTE, for example. In one embodiment, inthe absence of connectivity within the restaurant, the link selectelement 1105 may direct the COMMS 1102 to tether to a cellular equippeddevice corresponding to an order ID, such as devices 906 in FIG. 9, inorder to transmit acknowledgements and order state changes to thebackend server.

The connection monitor 1104 may provide connectivity status of theterminal 1100 to the order processor 1110 via bus CBUS. In oneembodiment, the order processor 1110 may generate order state changemessages from oldest to youngest update for each of the orders in thequeue 1121. Connectivity is maintained when the terminal 1100 receivesacknowledgement of a previously transmitted order state change messagefrom the backend server. Once acknowledged, the order processor 1110directs the state processor 1120 to delete the oldest state changeupdate for a corresponding order ID and shift pending updates so thatthe next oldest state change update becomes the oldest order update. Inone embodiment, state change updates are transmitted to the backendserver until its order state change record 1122 is empty, or untilconnectivity goes down.

As discussed above, the terminal 1100 may be configured as a fixed ormobile POS terminal, a kiosk, a host terminal, a management terminal, ora KDS, where configurations differ principally in size of thetouchscreen display 1103. As also discussed above, the camera 1114,microphone 1113, and motion sensor 1115 may be integrated in to theterminal 1100 or they may be coupled to the terminal 1100 viaconventional bus technologies such as a USB port, PCI port, etc. Theconfiguration parameters including order processing accelerationfeatures for the terminal 1100 are stored in the terminal ID 1109element, and the configuration parameters/features are accessed by aconfiguration manager 1111 within the order processor 1110 uponinitialization/reconfiguration of the terminal 1100. Thus, theconfiguration manager 1111 may generate images for display on thetouchscreen display 1103 and capture text and touches thereon, inaccordance with functions prescribed according to configurationparameters/features accessed from the terminal ID element 1109. Inaccordance with configuration parameters/features accessed from theterminal ID element 1109, the configuration manager 1111 may alsocapture audio from the microphone 1113, video from the camera 1114, andmotion from the motion sensor 1115, and may generate/receive messagesover MSG to/from the backend server for that are required to recognizecaptured images, keywords, and 3D gestures. Once received, theconfiguration manager 1111 may access the local gesture table 1131,which provides a mapping of recognized 3D gesture to commands to beexecuted by the terminal 1100 and may execute the commands via data sendto the touchscreen display 1103 over DATA. The configuration manager1111 may additionally access the local keyword table 1132, whichprovides a mapping of text (converted from audio by the backend server)to commands to be executed by the terminal 1100 and may execute thecommands via manipulation of the touchscreen display 1103, themicrophone 1113, the camera 1115, and/or the motion sensor 1115. Thelocal keyword table 1132 may also provide a mapping of keywords to menusuggestions, such as are discussed above with reference to FIG. 5, andmay send corresponding text/images over DATA for display on thetouchscreen display 1103.

Messages received from the communications circuit 1102 may also requireadditional functions to be performed by the terminal 1100. For example,when orders are placed by a browser-based or third-party based terminal,the backend server may transmit the order state change to the terminal1100 and the order processor 1110 may direct the state processor 1120 tocreate a corresponding order status record 1122 in the queue 1121.Similarly, when processing of transactions outside of the terminal'scapabilities (e.g., financial transactions with credit card providers,loyalty card discounts, etc.) are required, order processor 1110 maydirect the payment processor 1106 to generate messages to the backendserver to provide data (e.g., amounts, payment source type, cardswipe/chip information, etc.) to complete the transactions. Suchmessages are transmitted via COMMS 302. The payment processor 1104 mayfurther receive state changes (e.g., “order paid,” “payment source 1approved,” “discount amount,” etc.) to from the backend server and mayprovide these state changes to the order processor 1110 via TBUS. Theorder processor 1110 may then provide those updates to the queue 1121via OBUS. The terminal 1100 may further be employed to create an order.Accordingly, from order entry data received over DATA, the orderprocessor 1110 may direct the order initiation element 1107 to create anorder ID and may also direct the state processor 1120 to create acorresponding order state record 1122 in the queue 1121.

The terminal 1100 terminal according to the present invention isemployed to maintain a current state of all orders being fulfilled bythe restaurant. The current state of each of the orders is stored inorder current state fields 1112 within the order processor 1110. Theorder processor 1110 may also comprise domain specific rules, whichdefine actions required to synchronize conflicting order state changesbeing received and/or processed by the terminal 1100, where the domainspecific rules are unique to the restaurant. For example, suppose thatone terminal 1100 transmits a state change for a specific order to thebackend server, which is acknowledged by the backend server.Concurrently, a second terminal 1100 sends a state change for the sameorder. The backend server may respond to the second terminal with amessage indicating that the order current state field 1112 for the orderID within the second POS terminal 1100 state is out of date (due toconnection status) and providing the most recent current state of theorder. The domain specific rules are employed by the order processor1110 within the second terminal 1100 to merge the state changes local tothe second terminal 1100 and may store the merged state in thecorresponding current state field 1112. The order processor 1110 maythen direct the COMMS 1102 to transmit the corresponding (reconciled)state field 1112 contents to the backend server for distribution to allof the terminals 400 in the restaurant.

Advantageously, the present invention provides for improvements inperformance of computational resources within the terminals 1100 overthat which has heretofore been provided because the terminal 1100 may beemployed to process orders in the absence of network connectivity. Inaddition, unnecessary communications with the backend server areprecluded due to on-board conflict resolution logic. Moreover, computingperformance is increased because the terminal 1100 may be employed toprocess any of the other orders within the restaurant because thecurrent states of all restaurant orders are resident therein. Overallimprovements in the art are achieved through the synergistic andtailorable employment of facial recognition, voice commands, 3Dgestures, and suggested menu selections based upon detected menukeywords.

The terminal 1100 according to the present invention is configured toperform the functions and operations as discussed above. The terminal1100 may comprise digital and/or analog logic, circuits, devices, ormicrocode (i.e., micro instructions or native instructions), or acombination of logic, circuits, devices, or microcode, or equivalentelements that are employed to execute the functions and operationsaccording to the present invention as noted. The elements employed toaccomplish these operations and functions within the terminal 400 may beshared with other circuits, microcode, etc., that are employed toperform other functions and/or operations within the terminal 400.According to the scope of the present application, microcode is a termemployed to refer to a plurality of micro instructions. A microinstruction (also referred to as a native instruction) is an instructionat the level that a unit executes. For example, micro instructions aredirectly executed by a reduced instruction set computer (RISC)microprocessor. For a complex instruction set computer (CISC)microprocessor such as an x86-compatible microprocessor, x86instructions are translated into associated micro instructions, and theassociated micro instructions are directly executed by a unit or unitswithin the CISC microprocessor.

The terminal 1100 may be embodied as one or more central processingunits (CPUs) that are coupled to both transitory and non-transitorystorage (i.e., memory), where one or more application programs that areconfigured to perform the terminal functions discussed above are storedin the non-transitory storage, transferred to the transitory storage atrun time, and executed by the one or more CPUs.

Finally turning to FIG. 12, is a diagram 1200 is presented detailingexemplary update/status messages according to the present invention thatflow between a backend server and a terminal.

An order assignment message 1210 transmitted by the backend server toone or more terminals may comprise fields 1201 having a specificterminal ID TERMID assigned for a particular order ID ORDERID along witha table number TABLENUM having a given number of seats NUMSEATS. Themessage 1210 may further comprise a SPECIAL field 501 via which specialrequirements (e.g., high chair, wheel chair access) are communicated tothe terminal.

An order state change message 1220 transmitted from a terminal to thebackend server may comprise TERMID and ORDERID fields 1201 as describedabove, along with one or more groups of ITEM, MOD, and SEAT# fields1201, where contents of the ITEM field 1201 indicated a menu itemordered for a given seat number at the table along with anymodifications to the item number (e.g., rare, no onions, etc.).

A payment state change message 1130 transmitted from a terminal to theserver may comprise TERMID and ORDERID fields along with one or moregroups of PMTREQ, AMT, and TIP fields 1201, where contents of the PMTREQfield 1201 indicate a payment type (e.g., cash, MasterCard, etc.), andcontents of AMT and TIP indicate amount of payment for the particularpayment type along with a tip amount.

An order closeout message 1240 may comprise TERMID and ORDER ID fields1201 as noted above, along with a CLOSED field 1201, the contents ofwhich indicate whether the particular order ID is open or closed.

A terminal configuration message 1250 exchanged between a terminal andthe backend server may comprise a TERMID field 1201 as noted above alongwith a CONFIG field 501 and a plurality of order processing accelerationfeatures fields ACC1-ACCN 1201, where the contents of CONFIG specifyconfiguration parameters for the terminal (e.g., KDS, POS terminal,management terminal, delivery terminal, kiosk, etc.). The orderprocessing acceleration features fields ACC1-ACCN 1201 prescribespecific order acceleration features to be employed by the terminal suchas, but not limited to, 3D gestures, voice recognition, facialrecognition, and accelerated menu suggestions based upon recognizedkeywords.

A conversion message 1260 may comprise AUDIO, TEXT, MOTION, GESTURE,IMAGE, and ID fields 1201, where contents of AUDIO indicate capturedaudio that is transmitted to the backend server and contents of TEXTindicate text that has speech-to-text conversion has generatedcorresponding to the contents of AUDIO, and which is received by theterminal. Contents of MOTION indicate motions that are captured by themotion sensor that are transmitted to the backend server and contents ofGESTURE indicate a 3D gesture received from the server that correspondsto the contents of MOTION. Likewise, contents of IMAGE indicate an imagecaptured by the camera and which are transmitted to the server. Theserver in turn provides contents of ID, which provides an identity indexthat corresponds to the contents of IMAGE.

The messages 1210, 1220, 1230, 1240, 1250, 1260 are not exhaustive ofthose what may be employed according to the present disclosure but areprovided herein to teach further aspects and advantages according to thepresent invention.

Portions of the present invention and corresponding detailed descriptionare presented in terms of software, or algorithms and symbolicrepresentations of operations on data bits within a computer memory.These descriptions and representations are the ones by which those ofordinary skill in the art effectively convey the substance of their workto others of ordinary skill in the art. An algorithm, as the term isused here, and as it is used generally, is conceived to be aself-consistent sequence of steps leading to a desired result. The stepsare those requiring physical manipulations of physical quantities.Usually, though not necessarily, these quantities take the form ofoptical, electrical, or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, or as is apparent from the discussion,terms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, a microprocessor, a central processingunit, or similar electronic computing device, that manipulates andtransforms data represented as physical, electronic quantities withinthe computer system's registers and memories into other data similarlyrepresented as physical quantities within the computer system memoriesor registers or other such information storage, transmission or displaydevices.

Note also that the software implemented aspects of the invention aretypically encoded on some form of program storage medium or implementedover some type of transmission medium. The program storage medium may beelectronic (e.g., read only memory, flash read only memory, electricallyprogrammable read only memory), random access memory magnetic (e.g., afloppy disk or a hard drive) or optical (e.g., a compact disk read onlymemory, or “CD ROM”), and may be read only or random access. Similarly,the transmission medium may be metal traces, twisted wire pairs, coaxialcable, optical fiber, or some other suitable transmission medium knownto the art. The invention is not limited by these aspects of any givenimplementation.

The particular embodiments disclosed above are illustrative only, andthose skilled in the art will appreciate that they can readily use thedisclosed conception and specific embodiments as a basis for designingor modifying other structures for carrying out the same purposes of thepresent invention, and that various changes, substitutions andalterations can be made herein without departing from the scope of theinvention as set forth by the appended claims. For example,components/elements of the systems and/or apparatuses may be integratedor separated. In addition, the operation of the systems and apparatusesdisclosed herein may be performed by more, fewer, or other componentsand the methods described may include more, fewer, or other steps.Additionally, unless otherwise specified steps may be performed in anysuitable order.

Although specific advantages have been enumerated above, variousembodiments may include some, none, or all of the enumerated advantages.

What is claimed is:
 1. A user-adaptive order processing terminal,comprising: a display, configured to display electronic menu items in afirst area for selection by a user, wherein said electronic menu itemsare accessed via selection of one or more sub-menu items; a microphone,configured to detect audio spoken by said user; a configuration manager,coupled to said display and said microphone, configured to capture saidaudio from said microphone, to transmit said audio via first messages toa backend server, to receive second messages from said backend serverproviding text that corresponds to said audio, to access suggested menuitems that correspond to said text, and to modify a second area of saiddisplay to present said suggested menu items for selection, wherein saidsuggested menu items would otherwise be presented in said first areathrough selection of said one or more sub-menu items; and a motionsensor, coupled to said configuration manager, configured tosubsequently detect distance to and motions performed by said user,wherein said configuration manager captures movements of said user fromsaid motion sensor, transmit said motions via third messages to saidbackend server, receives fourth messages from said backend serverproviding 3-dimensional (3D) gestures that correspond to said movements,and adds one or more of said suggested menu items to an electronic orderin accordance with said 3D gestures.
 2. The user-adaptive orderprocessing terminal as recited in claim 1, further comprising: a localgesture table, coupled to said configuration manager, that provides amapping from 3D gestures to commands to be executed by the terminal,wherein said configuration manager executes said commands via data sendto said display.
 3. The user-adaptive order processing terminal asrecited in claim 2, wherein said display comprises a touchscreendisplay, and wherein the terminal is employed as a self-ordering kiosk,and wherein said user is not required to touch said display in order tocreate said electronic order.
 4. The user-adaptive order processingterminal as recited in claim 2, wherein said terminal further comprises:a state processor, configured to queue state changes in order queuesthat correspond to said electronic orders; and an order processor,coupled to said state processor and within which said configurationmanager is disposed, configured to generate said state changes, andconfigured to access and transmit said state changes in each of saidorder queues to said backend server, from oldest to youngest, whenoperably connected to a network, wherein said order processor comprisescurrent order state fields corresponding to all of said electronicorders.
 5. The user-adaptive order processing terminal as recited inclaim 4, wherein the terminal operably connects to said network via oneor more Wi-Fi access points.
 6. The user-adaptive order processingterminal as recited in claim 1, wherein said configuration managermodifies text size of said electronic orders on said display as afunction of said distance.
 7. The user-adaptive order processingterminal as recited in claim 1, further comprising: a local keywordtable, coupled to said configuration manager, that provides a mapping ofsaid text to said suggested menu items.
 8. A user-adaptive orderprocessing terminal, comprising: a display, configured to displayelectronic menu items in a first area for selection by a user, whereinsaid electronic menu items are accessed via selection of one or moresub-menu items; a microphone, configured to detect audio spoken by saiduser; an order processor, coupled to said display and said motionsensor, configured to maintain current states of all orders beingfulfilled, said order processor comprising: current state fields, eachmaintaining a current state for a corresponding one or said orders beingfulfilled; and a configuration manager, coupled to said display and saidmicrophone, configured to capture said audio from said microphone, totransmit said audio via first messages to a backend server, to receivesecond messages from said backend server providing text that correspondsto said audio, to access suggested menu items that correspond to saidtext, and to modify a second area of said display to present saidsuggested menu items for selection, wherein said suggested menu itemswould otherwise be presented in said first area through selection ofsaid one or more sub-menu items; and a motion sensor, coupled to saidorder processor, configured to subsequently detect distance to andmotions performed by said user, wherein said configuration managercaptures movements of said user from said motion sensor, transmit saidmotions via third messages to said backend server, receives fourthmessages from said backend server providing 3-dimensional (3D) gesturesthat correspond to said movements, and adds one or more of saidsuggested menu items to an electronic order in accordance with said 3Dgestures.
 9. The user-adaptive order processing terminal as recited inclaim 8, further comprising: a local gesture table, coupled to saidorder processor, that provides a mapping from 3D gestures to commands tobe executed by the terminal, wherein said configuration manager executessaid commands via data send to said display.
 10. The user-adaptive orderprocessing terminal as recited in claim 9, wherein said displaycomprises a touchscreen display, and wherein the terminal is employed asa self-ordering kiosk, and wherein said user is not required to touchsaid display in order to create said electronic order.
 11. Theuser-adaptive order processing terminal as recited in claim 9, whereinsaid terminal further comprises: a state processor, configured to queuestate changes in order queues that correspond to said electronic orders,wherein said order processor generates said state changes, and toaccesses and transmits said state changes in each of said order queuesto said backend server, from oldest to youngest, when operably connectedto a network.
 12. The user-adaptive order processing terminal as recitedin claim 11, wherein the terminal operably connects to said network viaone or more Wi-Fi access points.
 13. The user-adaptive order processingterminal as recited in claim 8, wherein said configuration managermodifies text size of said electronic orders on said display as afunction of said distance to said user.
 14. The user-adaptive orderprocessing terminal as recited in claim 8, further comprising: a localkeyword table, coupled to said configuration manager, that provides amapping of said text to said suggested menu items.
 15. A user-adaptiveorder processing method, comprising: displaying electronic menu items ina first area for selection by a user, wherein the electronic menu itemsare accessed via selection of one or more sub-menu items; employing amicrophone to detect audio spoken by the user; capturing the audio fromthe microphone, transmitting the audio via first messages to a backendserver, receiving second messages from the backend server providing textthat corresponds to the audio, accessing suggested menu items thatcorrespond to the text, and modifying a second area of the display topresent the suggested menu items for selection, wherein the suggestedmenu items would otherwise be presented in the first area throughselection of the one or more sub-menu items; and subsequently detectingdistance to and motions performed by the user by capturing movements ofthe user from a motion sensor, transmitting the motions via thirdmessages to the backend server, receiving fourth messages from thebackend server providing 3-dimensional (3D) gestures that correspond tothe movements, and adding one or more of the suggested menu items to anelectronic order in accordance with the 3D gestures.
 16. The method asrecited in claim 15, further comprising: mapping 3D gestures to commandsto be executed by the terminal, executing the commands via data send tothe display.
 17. The method as recited in claim 16, wherein the displaycomprises a touchscreen display, and wherein the terminal is employed asa self-ordering kiosk, and wherein said user is not required to touchsaid display in order to create said electronic order.
 18. The method asrecited in claim 9, wherein the terminal further comprises: via a stateprocessor, queueing state changes in order queues that correspond to theelectronic orders; and generating the state changes, and accessing andtransmitting the state changes in each of the order queues to a backendserver, from oldest to youngest, when operably connected to a network19. The method as recited in claim 15, further comprising: modifyingtext size of the electronic orders on the display as a function of thedistance.
 20. The method as recited in claim 15, further comprising: viaa local keyword table, mapping the text to the suggested menu items.